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Abstract 

Background: There are an increasing number of studies reporting the efficacy of educational strategies to facilitate 
the development of knowledge and skills underpinning evidence based practice (EBP). To date there is no 
standardised guideline for describing the teaching, evaluation, context or content of EBP educational strategies. The 
heterogeneity in the reporting of EBP educational interventions makes comparisons between studies difficult. The 
aim of this program of research is to develop the Guideline for Reporting EBP Educational interventions and 
Teaching (GREET) statement and an accompanying explanation and elaboration (E&E) paper. 

Methods/design: Three stages are planned for the development process. Stage one will comprise a systematic 
review to identify features commonly reported in descriptions of EBP educational interventions. In stage two, 
corresponding authors of articles included in the systematic review and the editors of the journals in which these 
studies were published will be invited to participate in a Delphi process to reach consensus on items to be 
considered when reporting EBP educational interventions. The final stage of the project will include the 
development and pilot testing of the GREET statement and E&E paper. 

Outcome: The final outcome will be the creation of a Guideline for Reporting EBP Educational interventions and 
Teaching (GREET) statement and E&E paper. 

Discussion: The reporting of health research including EBP educational research interventions, have been criticised 
for a lack of transparency and completeness. The development of the GREET statement will enable the standardised 
reporting of EBP educational research. This will provide a guide for researchers, reviewers and publishers for 
reporting EBP educational interventions. 
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Background 

Evidence Based Practice (EBP) is a universal philosophy 
for decision making in health care which considers the 
patient's perspective, therapist expertise and best avail- 
able research evidence [1]. It has been proposed that 
EBP consists of a five step process with the first four 
steps involving asking a clinical question, acquiring and 
appraising the evidence, and applying the evidence into 
clinical practice. The fifth step encourages individuals to 
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reflect upon the process undertaken in the first four 
steps [1,2]. 

The inclusion of EBP education in entry-level training 
is an accreditation requirement for many health profes- 
sional disciplines [3,4]. The Sicily consensus statement on 
EBP [2] recommended that EBP curricula should be 
grounded in the five step model, the efficacy and effective- 
ness of teaching each step should be researched (under- 
pinned by systematic reviews) and that courses claiming 
to teach EBP should evaluate each step using validated as- 
sessment tools. In response to the increasing number of 
instruments available to assess various aspects of EBP, a 
second Sicily statement has recently been published [4]. 
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This statement presents the classification rubric for EBP as- 
sessment tools in education' (CREATE) framework to pro- 
vide guidance and recommendations for developers 
of educational instruments. One of the intentions of the 
CREATE framework is to allow "comparisons across stud- 
ies by using a common set of outcome tools" [4]. The 
framework also provides information concerning classifica- 
tion of instruments based upon the five EBP steps, assess- 
ment category and intent, intended audience and pedagogy. 
The publication of a second Sicily statement reflects the 
rapid evolution of EBP and the need for standardised 
reporting processes. 

A number of systematic reviews of EBP educational 
interventions have commented on the lack of detail 
provided in the reporting of EBP educational interventions 
[5-9]. The value of standardised guidelines for reporting 
research processes is well accepted, with more than 200 
reporting guidelines currently available for health research 
[10]. These include randomised controlled trials (RCT) 
(n = 14) [e.g. the CONsolidated Standards of Reporting 
Trials (CONSORT)] [11], systematic reviews with or with- 
out meta analyses (n = 3) [e.g. Preferred Reporting Items 
for Systematic Reviews and Meta-Analyses (PRISMA)] 
[12], observational studies (n = 30) [e.g. Strengthening the 
Reporting of Observational Studies in Epidemiology 
(STROBE)] [13] and non-randomized trials (n = 7) [e.g. 
Transparent Reporting of Evaluations with Nonrandomized 
Designs (TREND)] [14]. To the best of our knowledge no 
reporting guideline exists which presents the key items to 
be considered when describing educational interventions 
for teaching foundation knowledge and skills of EBP. 

Recently, recommendations for the phases necessary to 
develop a reporting guideline in health research have been 
published [15]. The development process planned for this 
research program is based on these recommendations. 

The proposed reporting guideline for EBP educational 
interventions is planned to be used in conjunction with, 
rather than to replace reporting guidelines appropriate for 
the specific research design (CONSORT, STROBE, 
TREND etc.). This reporting guideline has the potential to 
benefit a number of education, health and research 
stakeholders. The guideline will provide detailed informa- 
tion to assist authors developing manuscripts of EBP edu- 
cational interventions, peer reviewers and journal editors 
when reviewing such manuscripts. The guideline could 
also result in increased consistency in reporting of educa- 
tional interventions, which could allow consumers of re- 
search to directly compare the educational interventions 
between studies and assist the undertaking and reporting 
of systematic reviews. Finally, the proposed guideline 
could be valuable for curriculum developers to consider 
when designing EBP courses. 

In this proposal, an EBP educational intervention 
incorporates the teaching of knowledge and skills 



recommended in the five steps of EBP (Sicily statement 1) 
[2]. It does not include educational interventions designed 
for advanced or practitioner knowledge of the best evi- 
dence underpinning specific health care interventions or 
management strategies. The proposed study does not 
intend to develop a guideline or recommendations for re- 
search approaches (study designs), for assessing the efficacy 
of EBP educational interventions, educational strategies for 
teaching, learning or assessing EBP, or the specific syllabi / 
curricula for teaching EBP. The aim of this research project 
is to develop a Guideline for Reporting of Evidence based 
practice Educational interventions and Teaching (GREET) 
statement and E&E paper. 

An explanation of the terms used in this proposal 

Doctoral Panel 

This four member panel consists of the Principal Investi- 
gator (AP) currently undertaking this program of research 
as part of a Doctor of Philosophy in Health Science (PhD), 
University of South Australia and the supervisory team 
consisting of the principal supervisor (MTW) and associ- 
ate supervisors (MPM, LKL). 

Expert Panel 

Experts with prior knowledge and experience in EBP 
educational theory, authors from the two pre-existing 
Sicily statements [2,4], authors responsible for the devel- 
opment of reporting guidelines and the dissemination of 
scientific information. 

Research Team 

The combined Doctoral and Expert Panels. 
Delphi participants 

Participants in the Delphi consensus survey. 
EBP educational intervention 

An educational strategy used to facilitate knowledge and 
skills in the five steps of EBP; asking a clinical question, ac- 
quiring and appraising the evidence, applying the evidence 
into clinical practice and reflecting upon the process under- 
taken in the first four steps (Sicily statement 1) [2]. 

Methods 

The research program will be undertaken in three 
stages: 

1. Systematic review of EBP educational interventions. 

2. Delphi survey to gain a consensus opinion on items 
to be considered within a standardised reporting 
guideline for EBP educational interventions. 

3. Development and pilot testing of the draft GREET 
statement and accompanying E&E paper. 
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An Expert Panel of five members has been convened to 
review and guide each of the three stages of the program. 
Members of this panel currently include Dr Paul Glasziou 
(co-author of Sicily 1), Dr Julie Tilson (author of Sicily 2), 
Dr David Moher (author of reporting guidelines) and Dr 
James Galipeau (founder of a centre to study scientific 
writing and publications), and Dr Marilyn Hammick 
[Consultant to Best Evidence Medical Education (BEME)]. 

The role of the Expert Panel will be to: 

1. Review and finalise the proposed research protocol. 

2. Monitor the systematic review, identify any further 
relevant articles for inclusion, review and contribute 
to the manuscript reporting the results of the 
systematic review (stage 1). 

3. Monitor and review the results of each round of the 
Delphi and contribute to the manuscript for 
disseminating the results of the Delphi survey (stage 2). 

4. Guide and facilitate the process used to develop and 
pilot test the GREET statement and E&E paper 
including the method of communication (email, 
internet site), likely number of meetings, method of 
feedback and method of resolving disagreements for 
drafting of the GREET statement (stage 3). 

5. Review, finalise and contribute to the dissemination 
of the GREET statement and E&E paper. 

Stage 1 - systematic review 

This systematic review aims to describe how EBP educa- 
tional interventions have been reported in controlled stud- 
ies investigating the efficacy of EBP training/education. 
The proposed systematic review question is: 

'What specific information has been reported when 
describing educational processes used in EBP 
educational interventions?' 



Databases and search terms 

A preliminary systematic review protocol was drafted 
based on previous systematic reviews in the area of EBP 
and education [6,8,16-19], the evidence-based practice 
guidelines for the peer review of electronic search strat- 
egies [20] and the guidelines provided in the PRISMA 
statement [12]. Modifications were made following peer 
review of the draft search strategy protocol by members of 
the international Centre for Allied Health Evidence 
(iCAHE) and two academic librarians [20]. The proposed 
databases and search terms (including all terms and med- 
ical subject headings relating to EBP, education and health 
professional disciplines) are presented in Table 1. The 
search aims to retrieve original studies of EBP educational 
interventions, limited to participants aged over 18 years 
without limitations placed on language of publication or 



publication date. At this stage no limits were placed upon 
study design. 

Information sources and search plan 

Studies will be identified by searching electronic databases 
and reviewing the reference lists of included studies to 
identify further studies. The full list of included studies 
will be reviewed by the Expert Panel to identify any further 
studies. 

Selection procedure 

The initial search will be conducted independently by 
two researchers (AP and a person independent to 
this study) using the prospectively developed search 
protocol detailing the specific search terms with ad- 
equate translation for each database [20]. The results of 
the two independently completed searches will be 
compared and any disagreements will be resolved by dis- 
cussion. If the protocol requires refinement, the process 
will be repeated. 

Inclusion / exclusion criteria 

Studies will be eligible for inclusion in this review if 
they are: 

• Published in peer reviewed journals irrespective of 
language of publication or publication date. 

• Primary controlled trials defined as studies including 
a separate group for comparison (e.g. controlled 
trials; randomised controlled trials) which report 
original data for an educational intervention specific 
to developing knowledge and skills of EBP. 

• Educational interventions within any teaching mode 
(face to face, online, group or individual) and must 
include at least one of the five steps of EBP. 

• Any level of health professional training 
(undergraduate, postgraduate, continuing education 
courses). 

• Secondary studies which meet the criteria defined in 
the PRISMA statement [12] to be classified as a 
systematic review or meta-analysis reporting 
primary studies of educational interventions specific 
to developing basic knowledge and skills of EBP 
(retained to assist identification of relevant primary 
studies). 

Studies will be excluded if they: 

• Describe or report evidence based guidelines or 
educational interventions specific to health 
conditions rather than educational interventions to 
develop skills and knowledge of EBP (for example, 
evidence based education for conservative 
management of hip osteoarthritis). 
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Table 1 Draft search strategy plan [6-9,16-19,21] 


Databases (search from 
inception) 


Search terms 


MEDLINE 

Academic Search Premier 

ERIC 

CINAHL 


(1) Health or medic* or nurs* or "al ied health" or physiotherap* or "physical therap*" or "occupationa therap*" or " 
speech therap*" or "speech patholog*" or diet* or nutrit* or "social work*" or psycholog* or podiatr* 

//II I - " // I 1 ££• " II ■ j_l X-lt I) _l j_l X-ll .1 s | ■ .ji 

or ambulance paramedic or ambulance officer or music therap or art therap or osteopath or chiropract or 
dentist* or dental* or optometr* or "medical radiation*" or "medical radiation science*" or radiograph*or pharmac* 
or "exercise physiolog*" AND 


Scopus 


(2) Instrument or questionnaire or survey or tool or data collection AND 


EMBASE 




Cochrane library 
Web of Science 


(3) EBP or evidence-based practice or EBM or evidence-based medicine or best evidence medical education or BEME 
or evidence-based nursing or research evidence or evidence-based health care or critical apprais* AND 


nformit - health database 


(4) Teach* or learn* or educat* or curricul* or student* or train* or course* or workshop* or journal club* 


EQUATOR network 


or program evaluat* 



* truncation symbol. 



• Focus on educational interventions for facilitating 
learning of statistical concepts without at least one 
of the five key EBP steps. 

• Report barriers, facilitators, attitudes, and 
behaviours relating to EBP without an educational 
intervention. 

• Are narratives, letters and books providing 
recommendations or strategies for teaching skills in 
EBP. 

• Describe or report the development of a survey 
instrument to gather information about EBP 
curricula/syllabi or courses without an EBP 
intervention to apply/test the instrument. 

Screening process 

To ensure the selection process is consistent, the Doc- 
toral Panel will collaboratively review the first 150 
records to determine which articles meet the inclusion 
criteria based on title and abstract. Any issues and/or 
disagreements will be discussed and resolved by consen- 
sus. The Principal Investigator will then review titles and 
abstracts and remove any studies that do not clearly 
meet the inclusion criteria. 

From the resultant list, further review will be conducted 
by two reviewers independently (AP and MTW) to deter- 
mine the inclusion/exclusion of studies. The full text of 
relevant studies will be sought where the abstract is un- 
available or where a decision about inclusion/exclusion 
cannot be made based on information provided in the ab- 
stract. The reference lists of included studies will be 
screened to identify additional publications. Disagreements 
between the reviewers will be resolved by a third independ- 
ent reviewer. The complete list of controlled or higher level 
studies will be provided to the Expert Panel to consider 
whether any studies meeting the inclusion criteria have 
been omitted and these will be included as required. 



Data extraction 

All included studies will be accessed as full text, retrieved 
and stored electronically. A data extraction sheet will be 
developed prospectively based upon the data items used by 
Shaneyfelt, Baum, Bell, Feldstein, Houston, Kaatz, Whelan 
and Green. (2006), Flores-Mateo and Argimon (2007), 
Lewis, Williams and Olds (2011), McEvoy, Williams and 
Olds (2011) and the Cochrane Handbook "Checklist of 
items to consider in data collection" [21] (Table 2). 
Data are intended to be extracted in six domains; 

1. Descriptive: (e.g. journal, year of publication, authors, 
professional discipline of authors, corresponding 
author, contact details, language of publication, key 
words, study design). 

2. Participants: (e.g. "Learners" level of education, 
professional discipline, "teachers" qualification, 
experience in EBP, sample size). 

3. Intervention mode and delivery: (e.g. face to face, on- 
line, group, individual, lecture, practical, journal club, 
discussion groups). 

4. Content: (e.g. presence and detail of the five steps of 
EBP). 

5. Evaluation: (e.g. methods and instruments). 

6. Confounding issues: (e.g. verbatim statements of 
issues confounding the EBP educational intervention 
or interpretation of the learning outcome). 

The data extraction sheet will be pilot tested for inter- 
rater reliability by two reviewers (AP and one member 
of the Doctoral Panel) with a random sample of 10% of 
included studies. For items where extraction or coding is 
inconsistent between reviewers (<80% agreement) [22], 
reviewers will convene to clarify discrepancies, the 
guidelines for data extraction will be revised and the 
data extraction sheet modified accordingly. The data 
items will then be double extracted for all included 
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Table 2 Proposed data extraction items (adapted from the Cochrane Handbook 2011) [21] 

Information to be extracted 
Eligibility Reason for inclusion / exclusion 

For studies retained in review 
Descriptive 

Participants 

Intervention mode and 
delivery 

Content 

Evaluation 

Confounding issues 



Journal, author(s), citation and contact details, professional discipline of authors, year of publication, corresponding 
author and contact details, language of publication, country study undertaken, key words, study design. 

Learners' level of education, professional discipline, previous EBP exposure, sample size (including prospective sample 
size calculation), randomisation of sample, age, instructors' qualification, experience in EBP. 

Type of intervention, duration, mode of delivery. 

Reference used for EBP, EBP steps described e.g. (ask, acquire, appraise, apply and assess). 

Name and type of assessment method (e.g. exam, assignment, FRESNO), psychometric properties, whether a named 
test was used and/or modified, randomisation of intervention. 

Verbatim statements of issues confounding the EBP educational intervention or interpretation of the learning 
outcome. 



studies [22]. On completion of data extraction, data will 
be compared between extractors for consistency, analysed 
and reported descriptively (frequency of items reported in 
each domain). 

Assessment of methodological quality and validity of 
included studies 

The aim of this systematic review is to describe how EBP 
educational interventions have been reported rather than 
describing the efficacy of EBP educational interventions. 
As such assessment of methodological quality or external 
validity (generalisability of study and/or results) is 
not warranted [23]. It is possible that lower level designs 
(i.e. not RCT/controlled trials) may provide different 
descriptions of the educational intervention and report 
different items. To explore this possibility, data will be 
extracted from a random selection of excluded studies 
with lower level designs (pre-post without control and 
narrative reviews) and compared with the higher level 
studies (RCT/controlled trials) used during the reliability 
process (n = 15). 

Strategy for data synthesis 

A narrative synthesis is planned. Descriptive synthesis of 
the findings from included studies will be structured 
around the six domains of Descriptive, Participants, 
Intervention mode and delivery, Content, Evaluation and 
Confounding issues. Summaries of the results from 
included studies will be provided and a list of all items 
reported by included studies will be compiled. The 
range, mean and standard deviation for reporting of 
items will be calculated for descriptive purposes. 

Outcomes 

At the completion of the systematic review a list will 
be compiled containing all reported information items. 
This list will be reported using the six domains of (1) 



Descriptive, (2) Participants, (3) Intervention mode and 
delivery, (4) Content, (5) Evaluation and (6) Confounding 
issues and will include each specific item and the fre- 
quency each item was reported. This list will be used in 
stage 2 to ensure that all information items identified by 
the systematic review are provided to the Delphi 
participants to determine their importance and relevance 
for reporting. 

The final outcome of the systematic review will com- 
prise submission of a manuscript to disseminate the 
findings of the systematic review. 

Stage 2 

Delphi survey 
Design and setting 

Ethics Ethical approval has been obtained from the Uni- 
versity of South Australia Human Research Ethics Com- 
mittee (protocol no. 25590) 

Design A Delphi survey will be conducted according to 
the checklist of 18 items described by Sinha et al. (2011) 
[24]. The Delphi process will comprise a series of four 
rounds of questionnaire, response and feedback until 
consensus is achieved [25]. The first round survey will 
commence with an open ended question and the three 
subsequent rounds will be undertaken providing feed- 
back from the previous round and inviting further 
responses from participants [24]. It is intended that the 
Delphi will be informed by the systematic review under- 
taken in stage 1. Any items identified in the systematic 
review that are not included in the Delphi list at the 
completion of the second round will be added as 'add- 
itional' items for the third round of the Delphi. This will 
allow a further two rounds of review by participants for 
both items volunteered by the group as well as add- 
itional items derived from the systematic review, before 



Phillips et al. BMC Medical Education 2013, 13:9 
http://www.biomedcentral.eom/1472-6920/13/9 



Page 6 of 1 1 



the completion of the Delphi survey. Consensus will be 
defined 'a priori' as recommended by Sinha, Smyth and 
Williamson (2011) [24]. There are no current inter- 
nationally accepted criterion standards to determine 
whether a consensus has been reached. For the purpose 
of this study consensus will be deemed to be achieved 
for each item of >80 per cent agreement indicating sub- 
stantial to excellent agreement [26]. 

Participants 

All corresponding authors of studies included in the sys- 
tematic review and the editors of the journals in which 
these studies were published (stage 1) will be invited to 
participate in the Delphi survey (likely maximum num- 
ber 110). 

Recruitment process Invitation and participation in the 
Delphi survey will be completed via email which will 
outline the aim, likely time commitment and process of 
the Delphi survey [24]. Those who do not respond to 
the initial invitation will be emailed seven and 14 days 
after the initial invitation. 

All participants will be allocated a random identifica- 
tion number for reporting and collation of the results. 
Demographic data regarding the participant's profession, 
qualifications and contact details will be recorded. 
Participants will be invited to provide their name and 
consent to be acknowledged as a member of the Delphi 
panel in presentations / publications arising from this 
research. All participants who accept the invitation to 
participate in the Delphi survey will be invited to 
complete each and every Delphi round, regardless of 
participation in the previous round unless they indicate 
withdrawal from the Delphi. 

Procedure The Delphi process will be conducted using 
an electronic survey format with embedded links to survey 
software (SurveyMonkey"). This software allows an un- 
limited number of participants, questions / responses and 
allows for export of responses to both Excel and SPSS 
formats. 



First round 

A brief preamble will be provided concerning the aim of 
the survey, definition of key terms, likely time commit- 
ment, plan for four rounds as well as the importance of 
completing all four rounds. As circumstances may have 
changed since initial recruitment, participants will be 
asked to contact the Principal Investigator (AP) if they 
wish to withdraw and will be removed from the list of 
participants. 



The first round will commence with an example of a 
description of an educational process for facilitating 
knowledge and skills in EBP followed by an open ended 
question. 

The proposed initial question is: 

"If you were reading a study which reported an 
educational process for facilitating foundation skills in 
evidence based practice (ask, acquire, appraise, apply 
and assess) what information about the 
INTERVENTION would you expect to be included? 

Please list ALL the items of information that you 
would expect to be included by authors to describe any 
evidence based practice educational intervention. 

There will be one reminder following each Delphi 
round which will be sent seven days after the dissemin- 
ation of the survey. Participants will be provided a fur- 
ther seven days to respond and the Delphi round will 
close 14 days after the initial survey is sent. 

All completed surveys will be analysed by the Principal 
Investigator. All responses will be downloaded verbatim 
to an Excel spread sheet and analysed using Excel and/ 
or SPSS. Specific items within text responses to the open 
question will be identified and allocated by the Principal 
Investigator (and reviewed by the Research Team) into 
the six domains planned for data extraction within the 
systematic review: (1) Descriptive, (2) Participants, (3) 
Intervention mode and delivery, (4) Content, (5) Evalu- 
ation and (6) Confounding issues. All items that are not 
able to be clearly assigned to a domain will be discussed 
by the Doctoral Panel and a consensus decision for their 
allocation reviewed by the Expert Panel [24]. Once con- 
sensus has been reached concerning the domains and in- 
dividual items within domains, frequency of responses 
will be calculated (per item for the total number of 
respondents). 

First round analysis 

A list of items within domains will be compiled. Fre- 
quencies will be calculated for all items and all 
participants. 

Second round 

Each item produced from the initial round will be 
allocated an identification number and randomised for 
frequency (within domains) [26]. To minimise the po- 
tential influence on the selection of items, the fre- 
quency with which specific items were provided by 
participants will not be reported to participants in the 
second round. 
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The randomly ordered list of items generated from the 
first round will be provided to all participants with 
instructions to rate the importance of each item on a 
Likert scale. A rating of zero meaning that the item is of 
limited importance and not required for reporting, up to 
a rating of 10 meaning that the item is of high import- 
ance and therefore essential for reporting. 

The instruction for each participant will be: 

"Here is a list of items which participants indicated 
should be reported when describing EBP educational 
interventions. For each item, please rate how 
important you think each item is for reporting an EBP 
educational intervention on the scale below. " 

After completing the rating exercise, participants will 
have the opportunity to provide further items by 
responding to the following questions: 

• If you rated any items <4 or >8, please provide a 
brief justification or reference to support your 
choice. 

• Are there any other items that you believe should be 
reported when describing EBP educational 
interventions or teaching? 

• Are there any comments you would like to add? 

Second round analysis 

The total number of completed surveys (number of 
participants) will be recorded and the rating for each 
item for each respondent downloaded to a spread sheet 
for descriptive analysis. 

Likert scores will be designated into four categories. 
These categories, based upon the ranges used to define 
agreement for Cohen's kappa, have been adapted for the 
Likert scale system on SurveyMonkey® (whole numbers 
only) [26]. 

The four categories according to Likert score (0-10) 
are: 

• 0-4: Low importance. Item not included. 

• 5-6: Moderate importance. Possible consideration 
for inclusion. 

• 7-8: High importance. Likely to require inclusion. 

• >8: Very high importance. Essential for inclusion. 

Descriptive statistics [percent agreement, mean score, 
SD, range, mean absolute deviation from the median 
(MADM)] for each item and each category will be 
computed. To be considered to have met the consensus 
criterion, 80 per cent or more respondents will need to 
rate an item's importance within the range for one cat- 
egory (low importance 0-4, moderate importance 5-6, 
high importance 7-8, very high importance > 8). 



Third round 

The list of items from the second round of the Delphi 
will be cross-checked against items derived from the sys- 
tematic review (stage 1). If there are items from the sys- 
tematic review which do not appear in the results of the 
second round survey, they will be added as 'additional 
items' for the participants to consider in round three. 
This entire list of items will be randomly ordered and 
will comprise the third round survey. 

Participants will be provided with descriptive feedback 
(percent agreement, mean, score, SD, range, MADM) for 
each item and each category along with any items which 
have achieved consensus. Any items that reach consensus 
will not be required for further comment and will be 
included in a separate page (screen) of the survey. All 
other items, including new items derived from participants 
from round two and additional items from the systematic 
review will be listed in random order. Participants will be 
asked to rate each item using the Likert scale. 

After completing the rating exercise, participants will 
have the opportunity to provide further items by res- 
ponding to the following questions: 

• If you rated any items <4 or >8, please provide a 
brief justification or reference to support your 
choice. 

• Are there any other items that you believe should be 
reported when describing EBP educational 
interventions or teaching? 

• Are there any comments you would like to add? 

Third round analysis 

This will replicate the analysis from the second round 
including the number of participants from the second 
round and the rating scores for each item and any items 
which have achieved consensus. 

Fourth round 

This Delphi process has been prospectively planned to 
have four iterations. Feedback from the third round 
results will be included as part of the survey. The infor- 
mation provided in the feedback will include a list of all 
items generated from the preceding round and which 
items (if any) have achieved consensus. Any items that 
reach consensus will not be required for further com- 
ment and will be included on a separate page. All other 
items, including new items derived from round three, 
will be listed in random order and participants will be 
invited to rate each item using the Likert scale. 

After completing the rating exercise, participants will 
be asked two final questions: 

• Would you be interested in reviewing the draft of 
the reporting guideline and associated document? 
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• If you are currently undertaking an EBP educational 
strategy and plan to submit this for publication, 
would you be willing to pilot test the draft 
guideline? 

Fourth round analysis 

This will replicate the analysis from the previous two 
rounds. Items which do not reach the pre-determined 
level of consensus (>80% agreement) will be categorised 
based upon their mean rating score (0-4: Low import- 
ance. Item should not be included; 5-6: Moderate im- 
portance. Possible consideration for inclusion; 7-8: High 
importance. Likely to require inclusion and >8 Essential. 
Item included). 

Outcome of fourth round 

At the completion of the fourth round, all items derived 
from either the systematic review or the Delphi survey 
will have been described as either 1) item met consensus 
for inclusion in the reporting guideline or 2) item did 
not meet consensus but could be considered for inclu- 
sion in the reporting guideline, 3) item did not meet 
consensus and mean rating score suggests it should not 
be included in the reporting guideline. 

This list of information items will form the basis for 
the content of the first draft of the GREET statement 
which will be compiled by the Research Team in stage 3. 

Stage 3 Development and pilot testing of the 
reporting guideline and explanation and 
elaboration document 

The development of an explanation and elaboration 
paper to accompany the reporting guideline is of 
vital importance and yet is a common omission in the 
development and operationalization of most reporting 
guidelines [15]. The role of the explanation and elabor- 
ation document is to provide the background, rationale 
and justification for the guidelines as well as to provide 
examples for users regarding what information should 
be included and how to report this information. A 
detailed explanatory Explanation and Elaboration (E&E) 
paper is planned to be developed concurrently with the 
reporting guideline. 

Procedure 

The Research Team will be convened via an email/website 
discussion to determine the proposed plan for the devel- 
opment of the GREET statement and E&E paper. The aim 
of this initial discussion will be to determine the best form 
of communication (email, internet chat site etc.), to nom- 
inate a chair person from the Expert Panel, to plan 
meetings, method of feedback (e.g. track changes), to pro- 
vide a structure and work plan for writing the draft and 
method for resolving disagreements. 



Following this initial meeting the Principal Investigator 
in collaboration with the Doctoral Panel will draft the ini- 
tial Guideline for Reporting EBP Educational interventions 
and Teaching (GREET) statement and E&E paper. The 
basis for each item will be described in the draft 
recommendations. This will include the origin of the item 
(systematic review or Delphi survey or both), the degree of 
consensus achieved in the Delphi survey and a brief ra- 
tionale for inclusion. The draft document will be provided 
to the Expert Panel who will be convened in a Skype con- 
ference consensus meeting. The meeting will provide the 
opportunity for the Research Team to discuss the items to 
be included, and the layout and design of the GREET 
statement and E&E paper. The GREET statement and 
E&E paper will then undergo subsequent review and feed- 
back via email or subsequent Skype conference call at the 
Expert Panels' discretion until the final draft is considered 
complete. The draft GREET statement and E&E paper will 
then undergo review and pilot testing for the content, lay- 
out and wording of the document [15]. 

Pilot testing of reporting guideline and explanation and 

elaboration document 

Participants 

Participants who have expressed interest at the comple- 
tion of the fourth round of the Delphi survey in either 
reviewing the draft GREET statement and E&E paper or 
trialling the documents to report an EBP educational 
intervention will be contacted. 

Procedure 

The final draft of the GREET statement and E&E paper 
will be distributed to all interested participants. The 
participants will fall into two groups. One group will fulfil 
an editorial role by reviewing the guidelines to provide 
comments and suggestions regarding the content, struc- 
ture, clarity and layout. The second group will include 
participants who have indicated that they are planning to 
report an EBP educational intervention and will be invited 
to trial the GREET statement and E&E paper to report 
their EBP intervention and to provide comments and 
feedback on the utility of these two documents. Partici- 
pant feedback including clarification of any difficulties will 
be recorded verbatim and considered for incorporation 
into the checklist revisions [15]. 

Publication plan 

• Publication 1: Study protocol. 

• Publication 2: Systematic review. 

• Publication 3: Delphi survey. 

• Publication 4 & 5: GREET statement and E&E paper 
planning for simultaneous publication [15]. 
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Discussion 

Over the past 25 years the philosophy and practice of 
EBP has evolved rapidly. This is reflected by the number 
of instruments developed to evaluate the efficacy of the 
teaching of EBP [8]. This rapid growth has resulted in 
heterogeneity of studies and inconsistencies in reporting 
of data. To address this problem, Simera, Moher, Hirst, 
Joey, Schulz and Altman (2010) [27] recommend that 
academic and other research institutions "promote and 
support accurate and transparent reporting of health re- 
search studies and the use of reporting guidelines. . ." 
Despite the recent rise in the number of reporting 
guidelines for health research, there are no reporting 
guidelines for educational interventions. The aim of this 
program of research is to develop a reporting guideline 
(GREET statement) and an explanation and elaboration 
(E&E) paper to enable the consistent and transparent 
reporting of EBP educational interventions and teaching. 

Following a recent systematic review of reporting 
guidelines for health research, Moher, Weeks, Ocampo, 
Seely, Sampson, Altman, Schulz, Miller, Simera, Grimshaw 
and Hoey (2011) produced a guidance statement for 
developers of reporting guidelines. This guidance statement 
forms the backbone for this program of research. To de- 
velop a reporting guideline for health research, Moher, 
Schulz, Simera and Altman (2010) recommend undertaking 
'initial steps! which include identifying the need for a guide- 
line and reviewing the literature, followed by 'pre-meeting 
activities' which refer to the preparation required for the 
next stage, the 'face to face consensus meeting' where con- 
sensus is achieved. This 'pre-meeting stage' will often in- 
clude a Delphi survey to generate a list of items to be 
further discussed at the 'face to face consensus meeting'. 
This is then followed by the 'post meeting activities' which 
includes the development of the reporting guideline, E&E 
document and pilot testing and publication strategy. The 
'post-publication activities' form the final stage in the devel- 
opment process and includes dealing with the feedback and 
criticism and updating of the guidelines. 

The inclusion of the Expert Panel representing the 
fields of EBP education, research guideline development, 
epidemiology and journal editing will guide the develop- 
ment and testing processes used in this study. Engaging 
an expert panel is not a requisite step in the develop- 
ment process for reporting guidelines, however it 
provides a safeguard to ensure the entire development 
process is transparent and comprehensive [15]. 

In this research protocol, the systematic review and 
Delphi consensus survey amalgamate the recommended 
'pre-meeting' and 'face-to face consensus meeting' stages 
[15]. While a 'face to face consensus meeting' allows for a 
captive audience and the potential for clearer communica- 
tion there are several possible limitations. The process 
can be time consuming and expensive, with previous 



reporting guidelines requiring 2-3 days for the initial face 
to face meeting and a recommended funding requirement 
of $75K for this meeting alone [15,28]. The travel required 
to attend a face to face meeting may preclude international 
participants thus limiting attendance and potentially intro- 
ducing personal bias [24]. For these reasons, this protocol 
plans to use a Skype conference meeting in lieu of the 
recommended 'face to face consensus meeting' to discuss 
the results of the Delphi consensus survey. The Delphi sur- 
vey is intended to be used as the primary means of achiev- 
ing consensus with the Skype conference to discuss and 
finalise the content and proposed layout of the GREET 
statement and E&E paper. 

With respect to the Delphi consensus stage, this re- 
search protocol is based upon the recommendations 
from a recent systematic review as the consensus method 
[24]. There are currently no internationally accepted cri- 
terion standards to determine the number and compos- 
ition of participants for a Delphi survey [24]. Previous 
Delphi surveys have used as few as 10 to as many as 1685 
participants [29]. Delphi surveys have previously been 
used in the development of nine reporting guidelines for 
health research [22,30] with the number of participants 
(reported by only seven studies) ranging from 11 (CON- 
SORT) [ll]to more than 50 (SQUIRE) [31]. The poten- 
tial number of participants planned for this program of 
research (likely maximum of 110 participants) is larger 
than has been used in previous guideline development 
because all corresponding authors of articles included in 
the systematic review (encompassing researchers and 
educators in the field of EBP education, statisticians 
and epidemiologists) will be invited to participate in the 
Delphi survey [15]. 

Traditionally the Delphi survey uses a minimum of 
three and a maximum of four rounds or until consensus is 
achieved [25]. The number of rounds used in Delphi 
surveys to date varies from as few as two [12,28,30] to as 
many as six [32]. Four rounds will be used in the proposed 
Delphi survey to maximise open ended survey responses 
in the first two rounds, and allow for additional items 
identified in the systematic review to be introduced in 
round three. Commencing the Delphi survey with an open 
ended question rather than using a pre-determined list of 
items will ensure that the researchers' views are not 
imposed upon the participants, thus enabling the 
participants to suggest all possible outcomes for consider- 
ation [24]. 

Previous studies have used a systematic review to gen- 
erate a pre-determined list of items to open a Delphi 
survey [24]. In the current program of research, a sys- 
tematic review will be used to inform the Delphi survey 
with a list of items identified in the review incorporated 
into the Delphi at the completion of the second round. 
This should provide sufficient transparency and time over 
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two further Delphi rounds to determine the opinion of the 
panel regarding the importance of the items and whether 
or not they should be included as reporting requirements. 

It is recommended that the level of consensus for the 
Delphi survey is both clearly defined and declared 'a priori' 
[24]. However there are no current internationally 
accepted criterion standards to determine whether a con- 
sensus has been reached [24]. Previous Delphi studies 
used in the development of reporting guidelines for health 
research do not report adequate detail to provide a guide 
for this study. Only two of the Delphi surveys used in the 
development of reporting guidelines for health research 
reported the level of agreement required to achieve con- 
sensus [30,33] and consensus was not declared 'a priori' 
for any of the Delphi surveys [22,30]. Consensus can be 
defined as a level of agreement and calculated using a 
mathematical formula or equation (e.g. percentage agree- 
ment, kappa analysis). However a consensus opinion is 
not necessarily the 'correct' opinion nor is it necessarily an 
accurate answer to a question [34]. For the purpose of this 
study consensus will be defined 'a priori' and deemed to 
be achieved for each item using descriptive statistics 
of >80 per cent agreement [26]. 

The importance of reporting guidelines are well 
recognised and the benefits of reporting guidelines that 
have been developed using robust and widely accepted 
methodologies include improving the standard and trans- 
parency of reporting, assisting peer reviewers and editors 
to strengthen manuscripts and potentially streamlining re- 
search funding applications [14,22]. The advantages of 
E&E documents, previously described as an 'essential 
element' in the development of reporting guidelines, are 
less widely recognised, and as a result, these documents 
have been largely overlooked by guideline developers. The 
E&E document provides the background and justification 
for the reporting guideline and can assist users by provid- 
ing examples of how to report information [15]. There are 
very few E&E documents available in comparison to the 
large number of reporting guidelines available, with fewer 
than 15 per cent of reporting guidelines providing an ac- 
companying E&E document [15]. The development of an 
E&E paper forms an essential part of the final stage of this 
program of research. 

To determine how useable documents are, requires 
testing in the real world and Moher, Schulz, Simera and 
Altman (2010) recommend considering a pilot testing 
process in the development of the reporting guideline. 
However most reporting guidelines do not include a 
pilot testing process, with the minority of reporting 
guidelines (n = ll, 13.6%) reporting a pilot testing 
process [22]. Pilot testing is planned for this program of 
research to determine whether the GREET statement 
and E&E paper are written at an appropriate level and 
whether they provide sufficient instruction for potential 



users. This will be achieved by gaining two perspectives 
from participants; an editorial perspective and a user 
perspective. The Delphi participants will be invited to 
review the documents and provide feedback with respect 
to the layout, wording and structure of the documents. 
Researchers will be invited to use the GREET statement 
and E&E paper to report their educational interventions 
and to provide feedback on the usability of the GREET 
statement and E&E paper. 

There are knowledge frameworks other than EBP, but 
this decision making paradigm is rapidly becoming part of 
accreditation requirements for entry-level and post gradu- 
ate health programs in educational institutions and various 
health care registration bodies. There is a rapidly growing 
body of research concerning educational practices used to 
teach and facilitate knowledge and skills in EBP, hence the 
consensus papers concerning what should be taught in 
EBP (Sicily 1) and the development of specific outcome 
instruments for use in assessing EBP foundation know- 
ledge, skills, behaviours and attitudes (Sicily 2). There are 
philosophical differences in terms of which frameworks 
are appropriate for exploring educational interventions 
and definitions of what evidence should be considered. Re- 
gardless of which framework is proposed or accepted, 
there is value in having a guideline for use when reporting 
the educational intervention. This guideline would allow 
consistency in describing the educational intervention ir- 
respective of the study design / methodological approach 
used by researchers. 

The development of the GREET statement and E&E 
paper will be the culmination of a development process 
that is both robust and applies widely accepted method- 
ologies. This will be the first reporting guideline for 
EBP educational interventions and as such is an import- 
ant milestone. As the reporting of EBP educational 
interventions is inconsistent at best, improving the 
standard of how educational research is conducted, 
reported and published is imperative. Developing the 
GREET statement will provide a reporting guideline for 
EBP educational interventions that can be used in con- 
junction with existing guidelines for research design to 
potentially benefit all stakeholders in EBP education: 
researchers, educators, editors, reviewers and students. 
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